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Abstract — In this paper, we demonstrate some applications 
of compressive sensing over networks. We make a connection 
between compressive sensing and traditional information the- 
oretic techniques in source coding and channel coding. Our 
results provide an explicit trade-off between the rate and the 
decoding complexity. The key difference of compressive sensing 
and traditional information theoretic approaches is at their 
decoding side. Although optimal decoders to recover the original 
signal, compressed by source coding have high complexity, the 
compressive sensing decoder is a linear or convex optimization. 
First, we investigate applications of compressive sensing on 
distributed compression of correlated sources. Here, by using 
compressive sensing, we propose a compression scheme for 
a family of correlated sources with a modularized decoder, 
providing a trade-off between the compression rate and the 
decoding complexity. We call this scheme Sparse Distributed 
Compression. We use this compression scheme for a general 
multicast network with correlated sources. Here, we first decode 
some of the sources by a network decoding technique and 
then, we use a compressive sensing decoder to obtain the 
whole sources. Then, we investigate applications of compressive 
sensing on channel coding. We propose a coding scheme that 
combines compressive sensing and random channel coding for a 
high-SNR point-to-point Gaussian channel. We call this scheme 
Sparse Channel Coding. We propose a modularized decoder 
providing a trade-off between the capacity loss and the decoding 
complexity. At the receiver side, first, we use a compressive 
sensing decoder on a noisy signal to obtain a noisy estimate 
of the original signal and then, we apply a traditional channel 
coding decoder to find the original signal. 

I. Introduction 

Data compression has been a research topic of many 
scholars in past years. However, recently, the field of com- 
pressive sensing, originated in (T), © and 0, looked at 
the compression problem from another point of view. In this 
paper, we try to make a bridge between compressive sensing 
which may be viewed as a signal processing technique 
and both source coding and channel coding. By using this 
connection, we propose some non-trivial practical coding 
schemes that provide a trade-off between the rate and the 
decoding complexity. 

Compressive sensing has provided a low complexity ap- 
proximation to the signal reconstruction. Information theo- 
retic has been mostly concerned with accuracy of the signal 
reconstruction under rate constraints. In this paper, we seek 
to provide new connections which use compressive sensing 
for traditional information theory problems such as Slepian- 
Wolf compression and channel coding in order to provide 
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mechanisms to explicitly trade-off between the decoding 
complexity and the rate. 

Some previous work investigated the connection between 
compressive sensing and information theory. For example, 
reference [4 | studied the minimum number of noisy measure- 
ments required to recover a sparse signal by using Shannon 
information theory bounds. Reference [5] investigated the 
contained information in noisy measurements by viewing 
the measurement system as an information theoretic channel 
and using the rate distortion function. Also, reference J6) 
studied the trade-offs between the number of measurements, 
the signal sparsity level, and the measurement noise level 
for exact support recovery of sparse signals by using an 
analogy between support recovery and communication over 
the Gaussian multiple access channel. 

In this paper, we want to investigate applications of 
compressive sensing on source coding and channel coding. 
First, in Section [Till we consider distributed compression 
of correlated sources. In this section, by using compressive 
sensing, we propose a compression scheme for a Slepian- 
Wolf problem with a family of correlated sources. The pro- 
posed decoder is a modularized decoder, providing a trade-off 
between the compression rate and the decoding complexity. 
We call this scheme Sparse Distributed Compression. Then, 
We use this compression scheme for a general multicast 
network with correlated sources. Here, we first decode some 
of the sources by a network decoding technique and then, 
we use the compressive sensing decoder to obtain the whole 
sources. 

Then, in Section [IV] we investigate applications of com- 
pressive sensing on channel coding. We propose a coding 
scheme that combines compressive sensing and random chan- 
nel coding for a high-SNR point-to-point Gaussian channel. 
We call this scheme Sparse Channel Coding. We propose 
a modularized decoder, providing a trade-off between the 
capacity loss and the decoding complexity (i.e., the higher 
the capacity loss, the lower the decoding complexity.). The 
idea is to add intentionally some correlation to transmitted 
signals in order to decrease the decoding complexity. At the 
receiver side, first, we use a compressive sensing decoder on 
a noisy signal to obtain a noisy estimate of the original signal 
and then, we apply a traditional channel coding decoder to 
find the original signal. 

II. Technical Background and prior work 

In this section, we review some prior results, in both 
compressive sensing and distributed compression, which will 



be used in the rest of the paper. 

A. Compressive Sensing Background 

Let X e M™ be a signal vector. We say this signal is 
fc-sparse if k of its coordinates are non-zero and the rest 
are zero. Define a = — as the sparsity ratio of the signal. 
Suppose $ is a n x n measurement matrix (hence, Y = $X 
is the measurement vector.). If $ is non-singular, by having 
Y and <£>, one can recover X. However, [2| and [3] showed 
that, one can recover the original sparse signal by having far 
fewer measurements than n. Suppose for a given n and k, 
Tji satisfies the following: 





ENCODER 


Mi 







777 > pk\og{n/k) 



(1) 



where p is a constant. Let S be the power set of 
{1, 2, m}, and S m represent a member of S with car- 
dinality 777 (e.g., S m = {1, 2, in}). A in x 7i matrix *&,s m 
is formed of rows of $ whose indices belong to S m . <&s m 
represents an incomplete measurement matrix. 

Reference [7| showed that, if m satisfies (Q~|i and &s m 
satisfies the Restricted Isometry Property (which will be 
explained later), having these m measurements is sufficient 
to recover the original sparse vector. In fact, it has been 
shown in Q that, X is the solution of the following linear 
programming: 



min ||X[[ £l 
subject to Y — $5 m X, 



(2) 



where \\-\\ci represents the C\ norm of a vector. 
Let us refer to the complexity of this optimization by 
CX™° lseless (m, n). We say *&s m satisfies the restricted 
isometry property (RIP) iff there exists < 8 k < 1 such 
that, for any vector V G K™, we have: 

(l-5 fe )||V||| 2 < ||* 5m V||2 2 < (l + 6 k )\\V\\l 2 . (3) 
Reference [7| showed if, 

h + hk + S 3 k < 1, (4) 

then, the optimization (O can recover the original fc-sparse 
signal. This condition leads to ([T). Intuitively, if <&s m satisfies 
the RIP condition, it preserves the Euclidean length of k 
sparse signals. Hence, these sparse signals cannot be in the 
null-space of $>s m . Moreover, any k subset of columns of 
$ < 5 m are nearly orthogonal if «&s m satisfies RIP. Most of 
matrices which satisfy RIP are random matrices. Reference 
[8 1 showed a connection between compressive sensing, n- 
widths, and the Johnson-Lindenstrauss Lemma. Through this 
connection, [9| provided an elegant way to find 5 k by using 
the concentration theorem for random matrices and proposed 
some data-base friendly matrices (binary matrices) satisfying 
RIP. 

Now, suppose our measurements are noisy (i.e., Y = 
tfr^X + Z, where ||Z|| J c 2 = e.). Then, reference [lj showed 
that if satisfies RIP (O and X is sufficiently sparse, the 
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Fig. 1. A depth one multicast tree with n sources. 



following optimization recovers the original signal up to the 
noise level: 



subject to 



|X|| £l 
|Y - *s„ 



(5) 



< e. 



Hence, if X* is a solution of (O, then ||X — X*|| £2 < 0e, 
where f3 is a constant. To have this, the original vector X 
should be sufficiently sparse. Specifically, [1| showed that, if 



S 3k + 35 4k < 2, 



(6) 



then, the results hold. We refer to the complexity of this 
optimization by CX™ S!, (m,ii). 

B. Distributed Compression Background 

In this section, we briefly review some prior results in 
distributed compression. In Section (Till we will use these 
results along with compressive sensing results, making a 
bridge these two subjects. 

Consider the network depicted in Figure [T] This is a depth 
one tree network with n sources. Suppose each source i, has 
messages referred by Mi, wishes to send to the receiver. To 
do this, each source i sends its message with a rate Ri (i.e., 
say mi has m messages of source i. This source maps rrij to 
{1, 2, 2 rnR *}.). Let R = YX=\ Ri- Accordin g t0 Slepian- 
Wolf compression ([10Q, one needs to have, 



R>H(M 1 ,M 2 ,...,M n ) 



(7) 



By using linear network coding, reference [11] extended 
Slepian-Wolf compression for a general multicast problem. 
Note that, the proposed decoders in [ 10 1 and [|1 1| (a minimum 
entropy or a maximum probability decoder) have high com- 
plexity ( Ifl2l ). For a depth one tree network, references Ifl3l , 
03), US), H6), El and COD provide some low complexity 
decoding techniques for Slepian-Wolf compression. However, 
none of these methods can explicitly provide a trade-off 
between the rate and the decoding complexity. For a general 
multicast network, it has been shown in 1 19 1 that, there is no 
separation between Slepian-Wolf compression and network 



coding. For this network, a low complexity decoder for an 
optimal compression scheme has not been proposed yet. 

III. Compressive Sensing and Distributed Source 
Coding 

In this part, we propose a distributed compression scheme 
for correlated sources providing a trade-off between the de- 
coding complexity and the compression rate. References iTTOl 
and ifTTl propose optimal distributed compression schemes 
for correlated sources, for depth-one trees and general net- 
works, respectively. However, their proposed compression 
schemes need to use a minimum entropy or a maximum 
probability decoder, which has high complexity (0~2))- On the 
other hand, compressive sensing provides a low complexity 
decoder, by using a convex optimization, to recover a sparse 
vector from incomplete and contaminated observations (e.g., 
[1|, [2|, |3| and [7|). Our aim is to use compressive sensing 
over networks to design compression schemes for correlated 
sources. 

Note that, our proposed techniques can be applied on 
different network topologies, with different assumptions and 
constraints. Here, to illustrate the main ideas, first we con- 
sider a noiseless tree network with depth one and with 
correlated sources. Then, in Section IIII-B1 we extend our 
techniques to a general multicast network. 

A. A one-stage tree with n-correlated sources 

Consider the network shown in Figure Q] which is a 
one-stage noiseless multicast tree network with n-sources. 
First, consider a case where sources are independent. Hence, 
the problem reduces to a classical source coding problem. 
Say each source is transmitting with a rate Ri. Denote 
R = Yl7=i Ri- I* i s well-known that the following sum-rate 
for this network is the minimum required sum-rate: 



n 

R>J2 H ( M i)i 
i=l 



(8) 



where Mi is the message random variable of source i. Let 
us refer to the complexity of its decoder at the receiver as 
CXi n dep(n), since sources are independent. 

Now, we consider a case where sources are correlated. We 
formulate the correlation of sources as follows: 

Definition 1. Suppose /ii } t represents a realization of the i th 
source message random variable at time t (i.e., a realization 
of Mi at time t). Hence, the vector /J* = {/^i,t, ••• ! Mn,t} is 
the sources ' message vector at time t. We drop the subscript t 
when no confusion arises. Suppose sources are correlated so 
that there exists a transform matrix under which the sources ' 
message vector is k-sparse (i.e., there exists a nxn transform 
matrix $ and a k-sparse vector fi t such that JT t = <&Jh' ■)■ Let 
S be the power set of {1, 2, n), and S m denote members 
of S whose cardinality are m. Suppose ^s m , defined as in 
Section \II-A\ satisfies RIP (O, where m satisfies (fJ]) for a 
given n and k. We have, lH,S m = *5 m Mt'> where Ht.s m is 
components of Jit whose indices belong to S m . We refer to 
these sources as k-sparsely correlated sources. 



If we know zero locations of /It' and if the transform 
matrix is non-singular, by having each subset containing k 
sources, the whole vector JT t and Jit can be computed. Hence, 
by the data processing inequality, we have, 

H(M il ,...,M ik )+nH b (a)=H(M 1 ,...,M n ), (9) 

for and 1 < i\ < 1% < ... < ik < n where is the 
message of source ij. Also, by [2], having m sources, where 
m satisfies (HJ for a given k and n, allows us to recover the 
whole vector Jit' and therefore, JH. Hence, 



H(M h ,...,M im ) = H(M u ...,M n ). 



(10) 



Example 2. For an example of k-sparsely correlated sources, 
suppose each coordinates of Jit is zero with probability 
\ — Oi, otherwise it is uniformly distributed over {1, 2, 2 R }. 
Suppose the entries of $ are independent realizations of 
Bernoulli random variables: 



with prob. \ 
— h= with prob. i 



(ID 



Thus, as shown in [9], 4?s m satisfies RIP (0. Also, we 
have, 

H(M 1 ,...,M n ) = nH b (a) + k(R+ 1) 

w nH h (a) + kR. (12) 

The entropy of each source random variable for large n 
can be computed approximately as follows: 



H(Mi 



R+Uog(k). 



(13) 



Note that, the individual entropies of fc-sparsely correlated 
sources are roughly the same. Now, we want to use compres- 
sive sensing to have a distributed source coding providing 
a trade-off between the compression rate and the decoding 
complexity. 

We shall show that, if sources are fc-sparsely correlated, 
if we have m of them at the receiver at each time t, by 
using a linear programming, all sources can be recovered 
at that time. Hence, instead of sending n correlated sources, 
one needs to transmit m of them where m « n. We assume 
that the entropies of sources are the same. Hence, each source 
transmits with probability 7 = — . Therefore, by the law of 
large numbers, for large enough n, we have m transmitting 
sources. For the case of non-equal source entropies, one can 
a priori choose m sources with the lowest entropy as the 
transmitting sources. 

In Table IIII-AI we compare four different schemes: 
« The first scheme which Theorem [3] is about, is called 
Sparse Distributed Compression with independent cod- 
ing (SDCIC). The idea is to send m sources from 
n of them, assuming they are independent (i.e., their 
correlation is not used in coding). In this case, at the 
receiver, first we decode these m transmitted messages 



TABLE I 

Comparison of Distributed Compression Methods of 
Correlated Sources 



Compression Methods 


Minimum Min-Cut Rate 


Decoding Complexity 


SDCIC 




CX inJcp (m) + CX™" d '"(m, n) 


Slepian-Wolf (SW) 


ff(Mi,...,M„) 


CX,„(n) 


Combination of SDC and SW 


H(Mi,...,M„) 


CX, w (m) + CX™" d '"{m,n) 


Naive Correlation Ignorance 




CXind^n) 



and then, we use a compressive sensing decoder to 
recover the whole sources. The decoding complex- 
ity is CX mdep {m) + CI™ sc,ess (m,n). The required 
min-cut rate between each receiver and sources is 

• In the second scheme referred in Table IIII-AI as the 
Slepian-Wolf coding, one performs an optimal source 
coding on correlated sources. The required min-cut rate 
between sources and the receiver is if (Mi, ...,M n ) 
which is less than ^" l = iH(Mi j ). However, at the 
receiver, we need to have a Slepian-Wolf decoder with 
complexity CX sw (n). 

• One can have a combination of sparse distributed com- 
pression and Slepian-Wolf compression. To do this, 
instead of n sources, we transmit m of them by using a 
Slepian-Wolf coding. The required min-cut rate would 
be H(M^,.,.,M im ) which by flS, H(M U ...,M n ) = 
H{M n ,...,M Zm ) m < J2"Li H ( M h)- At the receiver, 
first we use a Slepian-Wolf decoder with complexity 
CX sw (m) to decode the m transmitted sources' infor- 
mation. Then, we use a compressive sensing decoder to 
recover the whole sources. 

« The fourth method is a naive way of transmitting n 
correlated sources so that we have an easy decoder at 
the receiver. We call this scheme a Naive Correlation 
Ignorance method. In this method, we simply ignore the 
correlation in the coding scheme. In this naive scheme, 
the required min-cut rate is Y^i=i H{Mi) with a de- 
coding complexity equal to CXi n( i e p(n). Note that, in 
sparse distributed compression with independent coding, 
the required min-cut rate is much less than this scheme 

(Er=i#( M v) «T,tiH(M i )). 

The following theorem is about sparse distributed com- 
pression with independent coding: 



Theorem 3. For a depth one tree network with n-sources, 
k-sparsely correlated, the required min-cut rate between the 
receiver and sources in sparse distributed compression with 
independent coding (SDCIC) is, 

m 

R>J2 H ( M ii)> ( 14 > 

where m is the smallest number satisfies (0 for a given n 
and k and ij is the index of j th active source. Also, the 
decoding complexity of SDCIC method is CXi n( i ep (m) + 
CX™ iseless {m,n). 

Proof: For a given n (number of sources) and k (the 




Mi, M 2 ,...,M n 
1, M 2 ,...,M n 



1, M 2 ,...,M n 



Fig. 2. A general multicast network with n sources. 



sparsity factor), we choose m to satisfy ([T). Now, suppose 
each source is transmitting its block with probability 7 where 
7 = — . Hence, by the law of large numbers, the probability 
that we have m transmitting sources' blocks goes to one for 
sufficiently large n. Without loss of generality, say sources 
1, m are transmitting (i.e., active sources) and other 
sources are idle. If Ri > H(Mi) for active sources and zero 
for idle sources, active sources can transmit their messages 
to the receiver and the required min-cut rate between the 
receiver and sources would be R > Y^JLi^iM^), Let 
S m indicate active sources' indices. First, at the receiver, 
we decode these m active sources. The complexity of this 
decoding part is CXi n d ep (jn) because we did not use their 
correlation in the coding scheme. Hence, we have ~fij~s^, 
active sources' messages at time t at the receiver (in this case, 
jHj^ = fJ-m,t))- Then, by having ]M~sZ and usin S 

the sparsity of /It', the following optimization can recover 
the whole sources: 



mm llMt Iki 

subject to ^t,s m = $S m ~Pi'- 



(15) 



The overall decoding complexity is CX in( i ep (m) + 
CX™ iseless (m,n). " ■ 

B. A General Multicast Network with Correlated Sources 



In this section, we extend results of Section IIII-AI to a 
general noiseless multicast network. Note that, for a depth 
one tree network, references (13), 0, GD, IE), G2) and 
|[T8l provide some low complexity decoding techniques for 
Slepian-Wolf compression. However, none of these methods 
can explicitly provide a trade-off between the rate and the 
decoding complexity. For a general multicast network, it 
has been shown in |[T9l that, there is no separation between 
Slepian-Wolf compression and network coding. For this net- 
work, a low complexity decoder for an optimal compression 
scheme has not been proposed yet. There are few practical 
approaches and they rely on small topologies |20l . In this 
section, by using compressive sensing, we propose a coding 
scheme which provides a trade-off between the compression 
rate and the decoding complexity. 



Consider a multicast network shown in Figure [2] which 
has n sources and T receivers. If sources are independent, 
reference ||2TI showed that the minimum required min-cut 
rate between each receiver and sources is as ([8]). This rate 
can be achieved by using linear network coding over the 
network. Reference ifTTIl showed that random linear network 
coding can perform arbitrarily closely to this rate bound. Say 
^ -^Indepi 71 ) * s me complexity of its decoder. 

Now, consider a case when sources are correlated. Refer- 
ence ifTTl extended Slepian-Wolf compression result ( iflOl ') to 
a multicast network by using network coding. The proposed 
decoder is a minimum entropy or a maximum probability 
decoder whose complexity is high ( lfl2l ). We refer to its 
complexity by CX^{n). 

In this section, we consider sources to be fc-sparsely 
correlated. We want to use compressive sensing along with 
network coding to propose a distributed compression scheme 
providing a trade-off between the compression rate and the 
decoding complexity. 

In the following, we compare four distributed compression 
methods for a general multicast network: 

• In the first scheme, we use sparse distributed compres- 
sion with independent coding. With high probability, we 
have m active sources. Then, we perform network cod- 
ing on these active sources over the network, assuming 
they are independent. At the receiver, first we decode 
these m active sources and then, by a compressive 
sensing decoder, we recover the whole sources. The de- 
coding complexity is CX^ dep (m)+CX^ mseless (m, n). 
The required min-cut rate between each receiver and 
sources is Y^Li H{M ii ). 

• In the second scheme, we use an extended version 
of Slepian-Wolf compression for a multicast network 
( ll22l ). Reference l22l considers a vector linear net- 
work code that operates on blocks of bits. The re- 
quired min-cut rate between sources and the receiver 
is H(M%, M n ). At the receiver, one needs to have 
a minimum entropy or a maximum probability decoder 
with complexity CX r s L ^(n). 

• A combination of sparse distributed compression and 
Slepian-Wolf compression can provide a trade-off be- 
tween the compression rate and the decoding com- 
plexity. For example, instead of n sources, one can 
transmit m of them by using a Slepian-Wolf coding for 
multicast networks. The required min-cut rate would be 
H{Mi 1 , Mi m ). At the receiver, first we use a Slepian- 
Wolf decoder with complexity CX^(m) to decode m 
active sources. Then, we use a compressive sensing 
decoder to recover the whole sources. 

• In a naive correlation ignorance method, we simply 
ignore the correlation in the coding scheme. In this naive 
scheme, the required min-cut rate is H{Mi) with 
a decoding complexity equal to CX^ d (n). 

The following theorem is about sparse distributed com- 
pression with independent coding for a multicast network: 

Theorem 4. For a general multicast network with n-sources, 
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Fig. 3. A modularized decoder for a) compressive sensing with source 
coding, (b) compressive sensing with channel coding. 

k-sparsely correlated, the required min-cut rate between each 
receiver and sources in sparse distributed compression with 
independent coding is, 

m 

«>^//(A/ i .), (16) 
i=i 

where m is the smallest number satisfies (0 for a given 
n and k. Also, the decoding complexity is CX^ dep (m) + 
CX™ lseless {m,n). 

Proof: The proof is similar to the one of Theorem [3] 
The only difference is that, here, one needs to perform linear 
network coding on active sources over the network without 
using the correlation among them. At the receiver, first, 
these m active sources' blocks are decoded (the decoding 
complexity of this part is CX^ dep (m)). Then, a compressive 
sensing decoder (fT~5T > is used to recover the whole sources. 

■ 

Remark: In this section, we explained how compressive 
sensing can be used with distributed source coding. At 
the receiver side, first, we decode the active sources by a 
network decoding technique and then, we use a compressive 
sensing decoder to recover the whole sources. This high 
level modularized decoding scheme of sparse distributed 
compression is depicted in Figure [3}a. On the other hand, 
when we use compressive sensing for channel coding (which 
will be explained in detail in Section lPVT ). we switch these two 
decoding modules (as shown in Figure [3}b). In other words, 
first, we use a compressive sensing decoder on a noisy signal 
to obtain a noisy estimate of the original signal and then, we 
use a channel decoder to find the original message from this 
noisy estimate. 

IV. Compressive Sensing and Channel Coding 

In this section, we make a bridge between compressive 
sensing, which is more a signal processing technique, and 
channel coding. We consider a high-SNR Gaussian point- 
to-point channel depicted in Figure @] We propose a coding 
scheme that combines compressive sensing and random chan- 
nel coding with a modularized decoder to provide a trade- 
off between the capacity loss and the decoding complexity 
(i.e., the higher the capacity loss, the lower the decoding 
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Fig. 4. A Point-to-Point Gaussian Channel 

complexity.). We call this scheme Sparse Channel Coding. 
We add intentionally some correlation to transmitted signals 
to decrease the decoding complexity. This is an example of 
how compressive sensing can be used with channel coding 
and could be extended to other types of channels. 

Consider a point to point channel with additive Gaussian 
noise with noise power N and transmission power constraint 
P, shown in Figure [4] Suppose we are in a high SNR regime 
(i.e., ■& ^> 1). The capacity of this channel is, 



1 P 
C« -logf— ). 

2 hy N' 



(17) 



An encoding and a decoding scheme for this channel can 
be found in 1121 . As explained in [12|, at the receiver, one 
needs to use a m-dimensional maximum likelihood decoder 
where m, the code length, is arbitrarily large. We refer to the 
complexity of such a decoder as CX m i(m). Our aim is to 
design a coding scheme to have a trade-off between the rate 
and the decoding complexity. 

Suppose m is arbitrarily large. Choose n and k to satisfy 
©. 

Definition 5. A (2 mR , m) sparse channel code for a point- 
to-point Gaussian channel with power constraint P satisfies 
the following: 

• Encoding process: message i from the set 
{1,2, ...,2 m } is assigned to a m-length vector 
W t , satisfying the power constraint; that is, for each 
codeword, we have, 



rn 



(18) 



where W%j is the j coordinate of codeword Wj. 

• Decoding process: the receiver receives Y, a noisy 
version of the transmitted codeword Wj (i.e., Y = 
Wj + Z, where Z is a Gaussian noise vector with Ltd. 
coordinates with power N). A decoding function g maps 
Y to the set {1, 2, 2 mR }. The decoding complexity 
is CX ml (k) + CX™ is y(m, n). 

A rate R is achievable if P" 1 as m — > oo, where, 



2 m " 

P T = iE Pr(9(Y) + i\M 



and M represents the transmitted message. 



(19) 



Theorem 6. For a high-SNR Gaussian point-to-point channel 
described in Definition \5\ the following rate is achievable, 
while the decoding complexity is CX m i (k)+CX™° isy (m, n): 



(20) 



where C is the channel capacity M 71 ), a, j3 and 5k are 
parameters defined in Section I//-A1 and Hb(-) is the binary 
entropy function. 

Before expressing the proof, let us make some remarks: 

> if we have (HJ (i.e., m/n = palog(l/a)), the achievable 
rate can be approximated as follows: 



R 



-C 



-H b {a) (21) 



log(l/a) alog(l/a) 

where a — k/n is the sparsity ratio. The first term of 
( f2Tb shows that the capacity loss factor is a log function 
of the sparsity ratio. The term 2nog(T7ay^ fc ( a ) ^ s ^ e mte 
gain that we obtain by using compressive sensing in our 
scheme. Note that, since the SNR is sufficiently high, 
the overall rate would be less than the capacity. 

• The complexity term CX m i(k) is an exponential func- 
tion, while CX™° lsy (m,n) is a polynomial. 

Proof: We use random coding and compressive sensing 
arguments to propose a concatenated channel coding scheme 
satisfying (l20l i. First, we explain the encoding part: 

• Each message i is mapped randomly to a fc-sparse vector 
Xj with length n so that its non-zero coordinates are 
i.i.d. drawn from Af(0, mqifal P)- n is chosen to satisfy 
(O, for a given k and m. 

• To obtain Wj, we multiply Xjbyamx?i matrix (3?s m ) 
satisfying the RIP condition (0 (i.e., W, ; = $5 m Xi). 
We send Wj through the channel. 

In fact, it is a concatenated channel coding scheme |23|. 
The outer layer of this coding is the sparsity pattern of Xj. 
The inner layer of this coding is based on random coding 
around each sparisty pattern. To satisfy transmission power 
constraint P, we generate each non-zero coordinate of Xj 
by Af(0, w^j P), Note that, the encoding matrix $5 m is 
known to both encoding and decoding sides. 

At the receiver, we have a noisy version of the transmitted 
signal. Suppose message i has been transmitted. Hence, Y = 
Wj + Z = $5 m Xj + Z. The decoding part is as follows: 

• Since Xj is fc-sparse, first, we use a compressive sensing 
decoder to find X, such that ||Xj — Xj||| < (3mN, as 
follows, where f3 is the parameter of (O: 



(22) 



subject to — ||Y-*5 Xjlli < N. 
m 



We assume that the complexity of this convex optimiza- 
tion is smaller than the one of a maximum likelihood 
decoder. 

Since we are in a high-SNR regime, by having Xj, we 
can find the sparsity pattern of Xj. It gives us the outer 
layer code of message i. The higher the outer layer code 
rate, the higher the required SNR. We shall develop this 



argument with more detail later 
• Having the sparsity pattern of Xj, we use a maximum 
likelihood decoder in a fc-dimensional space (i.e., non- 
zero coordinates of Xj) to find non-zero coordinates 
of Xi (the inner layer code). The complexity of this 
decoder is denoted by CX m i(k). 
Before presenting the error probability analysis, let us 
discuss different terms of ( f2Qb . Since in our coding scheme, 
for each message i, we send a fc-sparse signal by sending a 
771-length vector, we have the fraction — before the capacity 
term C. The capacity term C in d20l > comes from the inner 
layer coding scheme. The term ^Hb(a) comes from the 
outer layer coding scheme. Note that, since a is a small 
number, Hb(a) is close to one. Also, the ratio ^ depends 
on the outer layer code rate. Here, we assumed the SNR 
is sufficiently high so that we can use all possible outer 
layer codes. The term ^ log( p^+g^ ) is because of the 
power constraint and the RIP condition. Note that, if one 
performs a time-sharing based channel coding, a rate is 
achievable with a decoding complexity CX m i(k). By using 
the compressive sensing, we obtain additional rate terms 
because of the outer layer coding with approximately the 
same complexity. 

We proceed by the error probability analysis. Define S$ 
as the sparsity pattern of X^ (i.e., Sy = if Xy = 0, 
otherwise, Sy = 1.). Also, define 5, as the decoded sparsity 
pattern of message i. Without loss of generality, assume that 
message 1 was transmitted. Thus, Y = $5 m Xi + Z. Define 
the following events: 



By the law of large numbers, we have Pr 



IXillL 



1 



3=1 

1 



/3m 



E i = {-\\X i -X 1 \\i a <^N} 
n 2 n 

E pi = {E x ^ Si}. 



(23) 



Hence, an error occurs when Eq occurs (i.e., the power 
constraint is violated), or E P1 occurs (i.e., the outer layer 
code is wrongly decoded), or E\ occurs (i.e., the underlying 
sparse signal of the transmitted message Xi and its decoded 
noisy version Xi are in a distance greater than the noise 
level), or one of E\ occurs while i ^ 1. Say M is the 
transmitted message (in this case M = 1) and M is the 
decoded message. Let £ denote the event M ^ M . Hence, 

Pr{£\M = 1) = Pr(£) 

3=2 

< Pr{E ) + Pr{E pi ) + Pr{El) 
i=2 

by union bounds of probabilities. We bound these proba- 
bilities term by term as follows: 



> 

rjV P + e) — >• as n — >• oo. By using the RIP condition 
0) and with probability one, we have, 



\*sM\l,<{l + h)\\M\c<rnP 



(25) 



Hence, Pr(Eo) — » as n — > oo. 

After using the compressive sensing decoder mentioned 
in 63, we have Xi such that i||Xi-Xi||2. 9 < ^N. 
Suppose this error is uniformly distributed over different 
coordinates. We use a threshold comparison to deter- 
mine the sparsity pattern Si. The probability of error in 
determining Si determines how high the SNR should be. 
Intuitively, if we use all possible outer layer codes (all 
possible sparsity patterns), we should not make any error 
in determining the sparsity pattern of each coordinate j 
(i.e., Sij). Hence, we need sufficiently high SNR for a 
given n. On the other hand, if we decrease the outer 
layer code rate, the required SNR would be lower than 
the case before. Here, to illustrate how the required SNR 
can be computed, we assume that we use all possible 
sparsity patterns. Say E Pjl is the event of making an 
error in determining whether the j th coordinate is zero 
or not. We say Sy = if Xy < r. Otherwise, Sy = 1. 
Hence, for a given threshold r, we have, 

Pr{E Pjl |Sy = 0) = q 



Pr(E Pjl |Sy = 1) = 1-2<X 



where </>(.) is the cumulative distribution function of a 
normal random variable. Hence, 



Pr(E pi ) = l-Pr(E c pi ) 



(26) 



1-1 



n 



»(!-«) 



P+^N 



Note that, one can choose r and ■£ large enough to have 
Pr(E pi ) arbitrarily small. 
By El, Pr{E{) is zero. 

At the last step, we need to bound Pr(Ej) for j ^ 1. 
We have, 



Pr(E 3 ) - Pr(S J =Si)Pr(^|S J =Si) 
+ PrCS^SOPr^-lS^Si) 
= a k (l~ar- k Pr(E J \E J ^E 1 ).(27) 

Note that, Pr(Ej\'Ej ^ Si) goes to zero in a high 
SNR regime. By bounding the noise power of non-zero 
coordinates by (3mN (the whole noise power) and using 
random coding argument in a A: dimensional space, we 



have, 

Pr{E^ =S 1 )<-— -\—ps> (28) 

\ l + /3{l+6 k ) N) 

where < indicates the right hand side is less or equal 
than the left one in an exponential rate. Therefore, by 
d27i > and (fJHJ, and doing some manipulations, we have, 

P(£ , 3 )<2~" (ffb(Q)+ ^ los( « T W^' )) , (29) 

for j = 2,...,2 mR . 
Thus, in a sufficiently high SNR regime, by d24l) and d29l ). 
we have, 

P e m = Pr(£) = Pr(£\M = 1) (30) 
<- 2 mR 2~ n( - Hb( - a ^ + % lo g ^(i+a fc ) £)) 
< 2 m(B ~™ Hb(QK ™ c ~^ log( « T W )) 

For any given e > 0, we can choose n large enough to 
have R < ±C + %H b (a) + ^ log( ^^y) - e achievable. 

■ 

V. Conclusions 

In this paper, we demonstrated some applications of 
compressive sensing over networks. We made a connection 
between compressive sensing and traditional information 
theoretic techniques in source coding and channel coding in 
order to provide mechanisms to explicitly trade-off between 
the decoding complexity and the rate. Although optimal 
decoders to recover the original signal compressed by source 
coding have high complexity, the compressive sensing de- 
coder is a linear or convex optimization. First, we investigated 
applications of compressive sensing on distributed compres- 
sion of correlated sources. For a depth one tree network, 
references ([ 1 3 1 - [ 18 1) provide some low complexity decoding 
techniques for Slepian-Wolf compression. However, none of 
these methods can explicitly provide a trade-off between the 
rate and the decoding complexity. For a general multicast 
network, reference ifTTI extended Slepian-Wolf compression 
for a multicast problem. It is showed in [19] that, there is no 
separation between Slepian-Wolf compression and network 
coding. For this network, a low complexity decoder for an op- 
timal compression scheme has not been proposed yet. Here, 
by using compressive sensing, we proposed a compression 
scheme for a family of correlated sources with a modularized 
decoder providing a trade-off between the compression rate 
and the decoding complexity. We called this scheme Sparse 
Distributed Compression. We used this compression scheme 
for a general multicast network with correlated sources. Here, 
we first decoded some of the sources by a network decoding 
technique and then, we used a compressive sensing decoder 
to obtain the whole sources. 

Next, we investigated applications of compressive sensing 
on channel coding. We proposed a coding scheme that 
combines compressive sensing and random channel coding 
for a high-SNR point-to-point Gaussian channel. We called 



this scheme Sparse Channel Coding. Our coding scheme pro- 
vides a modularized decoder to have a trade-off between the 
capacity loss and the decoding complexity. The idea is to add 
intentionally some correlation to transmitted signals in order 
to decrease the decoding complexity. At the decoder side, first 
we used a compressive sensing decoder to get an estimate of 
the original sparse signal, and then we used a channel coding 
decoder in the subspace of non-zero coordinates. 
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